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Brief history of benchmarking at NAS 

NAS Parallel Benchmarks (NPB) 

Goal: MMM Measure efficiency of modern (parallel) 
machines running scientific application 

Audience: M User: application engineer, researcher 
Buyer + vendor 

Contents: 5 kernels, 3 pseudo apps (implicit CFD) 

Approach: NPB1: Paper-and-pendl specs - *91 
NPB2: Source code implementations 
(F77/C/MPI) - '97 

PBN: Source code implementations 
(HPF/OpenMP/Java) - 99 
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NAS Parallel Benchmark codes 1 

Kernels: 

• EP 

Random-number generator 

MMMM 

• IS 

Integer sort 


• CG 

Conjugate gradient 


• MG 

Multigrid method for Poisson eqn 


• FT 

Spectral method (FFT) for Laplace eqn 

Pseudo 

• BT 

ADI; Block-Tridiagonal systems 

apps: 

• SP 

ADI; Scalar Pentadiagonal systems 


• LU 

Lower- Upper symmetric Gauss-Seidel 
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Measuring Grid performance 


NAS Grid Benchmarks (NGB) 

Goals: MM Measure efficiency + functionality of modem 

Grids running (distributed) scientific application 
(includes public network performance) 

Audience: Grid user: application engineer, researcher 
Audience: Grid software/infrastructure developer 

Contents: 4 compound tasks: 3 pseudo apps, 2 kernels, 
all from NPB 


Approach: NGB1: Paper-and-pendl specs - X)1 
NGB2: Source code implementations 

(NPB2/PBN + Globus/ Leg ion/CORBA/ 
Grid Engine/Condor/ Java ...) 
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Grid benchmark requirements 
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Grid benchmark construction 

• Tests computational aspects of environment 


■ Provide synthetic Grid app. for scientific computing: 

• Is representative of scientific computing tasks 


Data Flow Graph (DFG) coupling scientific codes 

• Uses basic Grid services 


■ Specify: 

• Is not intrusive (no throughput stress testing) 


^ abstract services: authenticate, create task, 

• Contains communicating processes 


communicate 

• Does significant communication 


^ problem sizes (classes): S, A, B, C, ... 

• Is verifiable (deterministic, not interactively steered) 


■ Do not specify: 

• Needs no initialization data files 


^ Mapping, scheduling, fault tolerance, data security 

Is fair (no favorite before dust settles) 


■ Measure and reoort turnaround time 
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Nodes of NGB Data Flow Graphs 

Modified NPB mesh-based codes SP, BT, LU, MG, FT: 

^ Well-accepted, well-studied, widely portable 
^ Solid verification procedures exist 
^ Parallel versions available: MPI/OpenMP/HPF/Java 
^ No data files required, but ... 

^ output of one NPB may be input for another 
Symbolize components of scientific apps: 

■ BT, SP, LU: scientific comps (flow solvers) 

■ MG: post- processing (data smoother) 

• FT: visualization (spectral analysis) 
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NGB Data Flow Graphs, Sample size 


Embarrassingly Distributed (ED) 


Helical Chain (HC) 
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NGB Data Flow Graphs, Sample size 


S 

Approximate rationale for NGB DFG classes 

Visualization Pipe (VP) Mixed Bag (MB) 

* S: for testing only; completes in seconds 

1 
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♦ A: reasonable size eng. problem; fits on modem SSI 

i-PT* mg'-j-sft 


machine without time sharing processes 

• B: challenging problem size for advanced SSI machine 

• C: won't fit on most advanced SSI machines 
Considerations: 

• Keep # verification values limited 

• Provide significant range of problem sizes 

• Complete in reasonable time (s-hrs on critical path) 

• Benchmarks non-converging, numerically stable 

• Ensure T(compute) ~ T(communicate) 
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Visualization cycle Unbalanced chain 
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No NGB class W 

Only verification in Report node 

LP -> HC. CP -* VP 








